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RESUME 

This article reviews active labour-market policies (ALIVIP), of which training is 
prominent. For about 20 years now, they have been one of the most important 
measures to combat unemployment and exclusion from the labour market. 
But is training a successful and efficient policy to reduce unemployment, 
compared to other types of ALIVIP? We draw some conclusions based on a re- 
view of evaluation results. We then make recommendations for designing as 
well as evaluating training. We underline the need for ALIVIP evaluations to de- 
velop an insight into ‘why' training works (or not). Finally, we advocate a new 
approach to ALIVIP and training programme evaluation: a systemic evaluation 
of the effectiveness of a programme through its relation, interaction and com- 
plementarity with other programmes, institutions and main stakeholders in vo- 
cational education and training, employment and production. 


Introduction 

The present article summarises the reviews carried out by Hujer et al. 
(2004) and Walsh and Parsons (2004) in the framework of the third Cede- 
fop report on vocational education and training research (Descy and Tes- 
saring, 2005). Studies reviewed in this report address systematic and sci- 
entific evaluations of active labour market policies, with a focus on train- 
ing. The studies provide a broad geographical coverage of ALMP evalua- 
tions in European countries up to 2003 ('), including EU-wide reports for 


H Delivery of the original reports to Cedefop; published in Descy and Tessaring, 2004a, 2004b. 
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mid-term and final European Social Fund evaluations as well as national 
reports reviewing the European employment strategy. Descy and Tessar- 
ing (2005), based on these and other publications, drew conclusions and 
recommendations for improving the effectiveness of active labour market 
policies, with a focus on training. 


Active labour market policies 

Persisting unemployment during the past decades led to growing aware- 
ness of the need to increase the effectiveness of employment and labour 
market policies. There is now broad consensus on implementing active meas- 
ures to complement passive ones. 

The term ‘passive measures’ designates legal rights and entitlements 
of unemployed people to receive unemployment and other related social 
benefits, including early retirement benefits. Active measures refer to more 
voluntary policies that foster reintegration into the labour market. They in- 
volve implementing various programmes targeted at specific groups to tack- 
le specific problems affecting them. 

Since the 1 980s, the trend in European labour-market policies has been 
to complement the distribution of unemployment and other benefits by ac- 
tive measures to reintegrate people more quickly into the labour market. 
Between 1985 and 2000, expenditure on active measures increased pro- 
gressively in various European countries, and in some countries equalled 
or even surpassed spending on passive measures. For example, in 2000 
Sweden and Greece spent equal amounts on active and passive labour- 
market policies while Italy was spending more on active measures. The re- 
maining EU countries still spent more money on passive measures although 
they were moving towards an equal spending pattern, compared to the sit- 
uation in 1985 (Hujer, 2004; Figure 1). The OECD (2003, p. 73), in its the- 
matic review of adult learning, also notes that common to all countries in- 
cluded in the review is the increased importance given to training and oth- 
er active measures rather than reliance on passive reception of unemploy- 
ment benefits. 

The emphasis given to training and retraining in ALMP can be seen by 
looking at the distribution of spending. On average, in terms of percentage 
of GDP, training is the type of active measure in which most money was 
invested in 2003 in the EU-1 5 (Figure 2). Considering the share of expen- 
diture by type of action in individual EU-1 5 Member States (Table 1), it ap- 
pears that training is the most significant part of ALMP spending in Germany, 
Ireland, Austria, Portugal, Finland and the UK. It is the second most impor- 
tant part of expenditure in Denmark, Greece, Spain, France, Italy and 
Sweden. 

Compared with EU-1 5 countries, central and eastern European coun- 
tries (before enlargement) tended to spend less on training and retraining 
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(Walsh and Parsons, 2004, p. 229). They favoured other policies, such as 
job creation through public work and subsidised jobs ( 2 ). 


Evaluating ALMP 

Given the increasing importance of ALMP in general and training in par- 
ticular, one would expect their evaluation to be a common concern for gov- 
ernments. Surprisingly, this is not necessarily the case in European coun- 
tries. Until recently, only a few European countries carried out rigorous eval- 
uations (Descy and Tessaring, 2005, pp. 36-42). This weak evaluation cul- 
ture is accompanied by a lack of appropriate data for evaluation. In con- 
trast, there is a long-standing tradition of evaluating labour market pro- 
grammes in Canada and in the US (which is why much of the literature em- 
anates from these countries). ‘A distrust deeply rooted in the North Amer- 
ican society toward all government actions, combined with strong empha- 
sis on the principle of individual responsibility, renders it virtually impos- 
sible to implement labour market programmes without an evaluation by in- 
dependent experts' (IZA, 2000, p. 2). Most evaluations of ALMP therefore 
stem from the US and Canada but also more recently from northern 
European countries (Belgium, Germany, Ireland, the UK and Nordic coun- 
tries). Evaluations are not widespread in central and southern European 
countries; they are mainly carried out either as a requirement for receiv- 
ing European social funds or in the framework of the European employ- 
ment strategy. 

ALMP effectiveness: what works? 

This section discusses the results of evaluations of programme partic- 
ipation on the labour market outcomes for individuals - mostly in terms of 
employment and earnings - rather than investigating the net effect of pro- 
grammes on total employment. 

Table 2 summarises results obtained by various researchers in evalu- 
ating ALMP. It is difficult to compare the relative impact of each type of meas- 
ure but it is worth noting that: 

• special youth measures do not appear successful, unless this type of 
intervention is carried out early and is of broad scope; 

• job search assistance appears to help most unemployed people, even 
if some conditions need to be in place; 

• subsidies to employment also seem effective, contrary to direct job cre- 
ation. 


( 2 ) However, this may be because these active policies were recently introduced and the full set 
of ALMP was not yet available. For instance, some of these countries spent a large propor- 
tion of their budget developing their employment services in the 1990s. Once the required 
level of service was established, they started focusing on other kinds of employment poli- 
cies. 
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Figure 1 . Spending on active and passive labour-market policies, EU-1 5, 1 985, 
2000 (% of GDP) 
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Figure 2. Public expenditure on ALMP, EU15, 2003 (% of GDP) 
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Source: Eurostat, LMP data collection, EC 2005 


Table 1. Composition of active labour-market expenditure by type of action, EU-15, (%), 
2003 



Training 

Job rotation 
and job 
sharing 

Employment 

incentives 

Integration 
of the 
disabled 

Direct 

job 

creation 

Start-up 

incentives 

EU15 

39.4 

0.3 

19.7 

16.3 

19.4 

4.9 

BE 

17.9 

- 

21.5 

11.5 

48.6 

0.5 

DK 

33.8 

- 

31.8 

34.3 

0.1 

- 

DE 

46.8 

0.1 

12.3 

16.1 

15.1 

9.5 

IE 

36.5 

0.0 

22.2 

6.2 

35.1 

0.0 

EL 

27.9 

- 

16.6 

19.6 

- 

35.8 

ES 

20.0 

1.3 

43.0 

11.9 

16.2 

7.7 

FR 

36.9 

- 

10.3 

10.6 

41.8 

0.4 

IT 

36.8 

0.0 

50.3 

1.0 

4.2 

7.6 

NL 

21.0 

0.0 

3.2 

50.0 

25.9 

- 

AT 

64.6 

0.0 

13.1 

12.1 

9.2 

1.0 

PT 

53.0 

0.0 

28.9 

10.2 

7.3 

0.6 

FI 

47.4 

8.2 

17.3 

13.4 

12.0 

1.5 

SE 

37.5 

0.7 

14.2 

44.1 

- 

3.5 

UK 

81.2 

- 

1.9 

12.6 

4.0 

0.3 


NB: Luxembourg: no data available for the ‘training’ category. 
Source: Descy and Tessaring (2005) based on Eurostat, 2002. 



Combating labour market exclusion: does training work? 

Pascaline Descy, Manfred Tessaring 69 


Grubb and Ryan (1999, pp. 80-92) conclude from their review of eval- 
uation studies in the US that training for disadvantaged adult men and women 
yielded only modest gains in earnings, while training for disadvantaged youths 
showed no positive and even some negative impact. In Europe, training tends 
to increase the employability of disadvantaged participants but not their earn- 
ings. Again, the efficacy for young workers is uncertain. In addition, even 
if training does have an effect on participants’ employment probability, the 
aggregate employment effect appears to be weak. '[...] public training of- 
ten fails, particularly when it involves short, low-cost courses of remedial 
training and retraining, and when the criterion of success is a lasting gain 
in earning power, and not simply a short-term increase in employment rates 
for participants who continue to inhabit low-skilled labour markets. At the 
same time, public training can work when it sets its sights higher, aims at 
occupationally relevant needs in shortage labour market and takes train- 
ing quality seriously’ (ibid., p. 92) 

From their review of microeconometric studies in selected European coun- 
tries, Hujer et al. (2004) conclude that, in many instances, training programmes 
have positive effects; they found fewer examples of positive effects of oth- 
er types of ALMP, especially job creation programmes. Equally, Fay 
(1996, cited in Walsh and Parsons, 2004, p. 232) puts forward that expec- 
tations of the return of training measures for individuals should be modest. 
Nevertheless, small-scale, targeted programmes that reflect the needs of 
both employers and job-seekers offer the best prospective outcomes. 

Martin and Grubb (2001, p. 15) conclude that although training tends 
to be the most expensive active measure its efficacy is not obvious (at least 
in terms of traditional outcomes such as employment and earnings). Some 
programmes have yielded low or even negative rates of return for partici- 
pants when the estimated programme effects are compared with the costs. 

Selected studies carried out in northern European countries (summarised 
by Descy and Tessaring, 2005, p. 176-177) show it is difficult to draw a clear 
picture of the effectiveness of training from evaluation results (Table 2). Ef- 
fectiveness tends to vary across studies, according to external labour mar- 
ket circumstances and across target groups. 

Nevertheless, the following recommendations can be made for design- 
ing and implementing training programmes to increase the effectiveness 
of training in ALMP ( 3 ): 

• they should be targeted at specific groups and objectives (e.g. the long- 
term unemployed, unemployed youths, women returning to employment, 
illiterate adults, etc.); 

• they should allow time for active job search (which should also be ac- 
tively promoted); 

• they should be kept relatively small in scale and not be used as a large- 
scale solution to unemployment; 


( 3 ) Based on the following literature: Fay (1996, cited in Walsh and Parsons, 2004), Martin and 
Grubb (2001 : 1 4), Walsh and Parsons (2004), Hujer et al. (2004), Grubb and Ryan (1 999). 
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• they should lead to qualifications or certificates not only recognised but 
also valued on the labour market (involving employers in programme 
design and implementation seems to play a positive role); 

• they should establish strong links with local employers and provide work 
experience, while avoiding displacement or substitution effects; 

• when targeted at the young, they should be considered in association 
with general policies on education. 

Evaluations carried out in the framework of EU policies 

As indicated above, European countries do not all share the same eval- 
uation tradition. However, the necessity of evaluating projects financed by 
European social funds as well as of reviewing progress in the framework 
of the European employment strategy has led all EU Member States to pro- 
ceed to some form of evaluation or monitoring. 

The European employment strategy (EES) 

Concerns about the high level of unemployment in the EU led to the EES, 
a framework for a coordinated policy response which improves labour mar- 
ket policies while providing a common European framework for combating 
unemployment. Training is an important aspect of this strategy. This sec- 
tion presents selected results of the EES 2002 impact evaluation. 

The EES is a form of ‘management by objectives'. It is based on peer 
review and benchmarking policy and practice between Member States. The 
Council of the EU, following a proposal from the European Commission, 
decides on employment guidelines each year. These guidelines have to be 
transformed into national policies and reported on in national action plans 
(NAPs). NAPs are then assessed with a view to setting the next annual guide- 
lines. Since 2000, in addition to the employment guidelines, the Commis- 
sion issues specific recommendations to Member States. The requirement 
of reviewing labour market policies regularly is embedded in the EES. Thus, 
the EES contributes to developing and establishing evaluations and a com- 
mon learning culture in EU Member States. In 2001/02, Member States car- 
ried out a thorough review covering the first three years of the EES (from 
1998 to 2001). This provided a useful insight into active employment poli- 
cies (Table 3), and served as a basis for the 2002 Impact evaluation of the 
EES (European Commission, 2002). 

Evaluations carried out for the European Social Fund (ESF) 

The ESF provides funding for programmes which develop or regener- 
ate people’s employability. The ESF supports measures which aim to pre- 
vent and combat unemployment, develop human resources and foster in- 
tegration into the labour market, promote a high level of employment, equal 
opportunities for men and women, sustainable development and econom- 
ic and social cohesion. Programmes are planned by Member States with 
the European Commission and then implemented through a wide range of 
provider organisations in both public and private sectors ( 4 ). 
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Funding requirements and implementing the ESF also play a role in de- 
veloping an evaluation culture throughout the EU. The ESF requires eval- 
uation at three levels: 

(i) project level: each funded project is required to have an explicit moni- 
toring and evaluation strategy; 

(ii) national level: in the 1990s, Member States were involved in mid-term 
and final evaluations of their national ESF programmes; 

(iii) European level: the European Commission, drawing on national reports, 
indicates EU-wide trends. 

Table 4 summarises some findings of ESF evaluations. From the EU- 
wide report, it appears that targeted policies in general work better than non- 
targeted ones and that linking training to labour market and employer needs 
is always likely to improve their impact. 

Conclusions on evaluations carried out in the framework of EU policies 
Although the EES and the ESF contribute towards raising the profile of 
and using evaluations, benchmarking and peer reviews, there are still sub- 
stantial differences across EU countries in rigorous evaluation methods. Over- 
all, evaluations carried out in the framework of EU-supported policies dis- 
play the same pictures as more traditional evaluations of ALMP: there is 
no consistent indication of the effectiveness of training when it is get peo- 
ple back into employment. Training seems to have a positive effect but it 
tends to be modest and linked to prevailing labour market conditions. Train- 
ing needs to be targeted and it works better if employers are involved or if 
their needs are directly considered. In difficult labour market conditions, train- 
ing might not be sufficient to get people back into employment. 

Recommendations for evaluating training within ALMP 

Reviewing evaluation results tells us that the effects of training on em- 
ployment and earnings tend to be modest: it cannot be effective for all tar- 
get groups, regions and episodes of the business cycle. At the same time, 
it is a very costly type of ALMP. Consequently, does training work well enough, 
or are other kinds of ALMP preferable? We argue that whether an interven- 
tion is successful or not in increasing employment probability and earnings 
for participants is not sufficient to inform policy-making and design better 
and more sustainable interventions. 

Two main conclusions emerge from examining various attempts to eval- 
uate the effectiveness of training in ALMP: 

• current evaluation results do not provide a clear picture of the effective- 
ness of training; it tends to vary across programmes and target groups 
as well as across studies; 


( 4 ) This brief description of the ESF is based on the DG Employment and Social affairs ESF 2000- 
06 website: http://europa.eu. int/comm/employment_social/esf2000/introduction-en.htm#key. 
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Table 2. Lessons from evaluation literature on the effectiveness of ALMP, composite studies 


Appears 
to help 

Appears 
not to help 

General observations 

Source 

Training/retraining 

Disadvantaged 
adults, especially 
women 

Disadvantaged 
young people (?) 

Type of training: remedial training for disadvantaged wor- 
kers; 

mainstream programmes in the US: greatest gains in sub- 
sequent earning for adult women. Young register no gain 
or losses. 

in Europe, increases employability but tends to leave ear- 
nings unchanged; 

the picture does not seem as negative concerning the ef- 
fect on youth in Europe as it is in the US. 

Grubb and Ryan 
(1999) 

Long-term unem- 
ployed, in particular 
women 


Small positive effect on employment and earnings, 
which depends to a great extent on the business cycle 

Darand Tzannatos 
(1999) (i) 

Those laid off en 
masse 


Has a high dead-weight effect; most successful if on small 
scale and targeted towards the most vulnerable groups. 

Darand Tzannatos 
(1999) (i) 


Young people 

Did no better than control groups in terms of improving 
employment probability and earnings; social rates of re- 
turn tended to be negative. 

Darand Tzannatos 
(1999) (i) 

Men and women 
(young and middle- 
aged) 


The impact is higher for women than men. The impact is 
lower for older participants than the young. Those with 
primary and secondary education benefit more than 
those with post-secondary education. 

Fretwell et al. 
(1999) ( 2 ) 

Short-term unem- 
ployed 



Fretwell et al. 
(1999) ( 2 ) 

Unemployed 


In central and eastern Europe countries. With some limi- 
tations imposed by the general state of the labour mar- 
ket, in these countries, training emerges as a relatively low- 
cost measure. 

Walsh and Parsons 
(2004) 

of which: formal classroom training 

Women reentrants 

Prime-age men and 
older workers with 
low initial education 

Important that courses signal strong labour market rele- 
vance or signal high quality to employers; keep program- 
mes relatively small in scale. 

Martin and Grubb 
(2001) ( 3 ) 

On-the-job training 

Women re-entrants; 
single mothers 

Prime-age men (?) 

(?) Because some programmes gave positive results and 
others not; 

Must directly meet labour market needs. Hence, need to 
establish strong links with local employers, however in- 
creasing the risk of displacement. 

Martin and Grubb 
(2001) 
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Appears 
to help 

Appears 
not to help 

General observations 

Source 

Special youth measures (training, employment subsidies, direct job creation measures) 


Disadvantaged 

youths 

Effective programmes need to combine an appropriate 
and integrated mix of education, occupational skills, work- 
based learning and supportive services to young people 
and their families; early and sustained interventions are 
likely to be most effective; need to deal with negative at- 
titudes to work among the young. Adult mentors can help. 

Martin and Grubb 
(2001) 

young people 
to find a job 

young people 
to get higher pay 

Insertion programmes; evidence studies in France and in 
the UK; gains for young people result in displacement for 
other workers. 

Grubb and Ryan 
(1999) 

Job-search assistance (job clubs, individual counselling, etc.) 

Most unemployed 
but in particular, 
women 

and sole parents 


Must be combined with increased monitoring of the job 
search behaviour of the unemployed and enforcement of 
work tests 

Martin and Grubb 
(2001) 

of which: reemployment bonuses 

Most adult unem- 
ployed 


Requires careful monitoring and controls on both recipients 
and their former employers. 

Martin and Grubb 
(2001) 

Subsidies to employment 

Long-term unem- 
ployed; women re- 
entrants 


Require careful targeting and adequate control to maxi- 
mise net employment gains and social benefits. Some sub- 
stitution mechanisms might reduce net employment 
gains. 

Martin and Grubb 
(2001) 

Aid to unemployed starting enterprises 

Men (below 40, 
relatively better 
educated) 


Only works for a small subset of the population. 

Martin and Grubb 
(2001) 

Direct job creation 


Most adult 
unemployed 

Typically provides few long-term benefits and principle of 
additionality usually implies low marginal-product jobs; 
no significant or even negative effects in France, Switzer- 
land, Sweden and the UK. 

Martin and Grubb 
(2001) 

Hujer et al. (2004) 
( 4 ) 


NB: Cited in Walsh and Parsons (2004). 

( 2 ) Based on a review of evaluations up to 1 999 in four transitional countries: the Czech Republic, Hungary, Poland and Turkey. 

( 3 ) Martin and Grubb (2001 ) based their finding on: US Department of Labor (1 995), Fay (1 996), Friedlander et al. (1997), Grubb (1 995), HRDC 
(1997), Lerman (1997) and OECD (1993). 

( 4 ) Based on Lalive et al. (2000), Gerfin and Lechner (2000), Bonnal et al. (1997), Brodaty et al. (2001), Firth et al. (1999); Payne et al. (1996) 
and Larsson, (2000). 


Source: This table is based on an original table from Martin and Grubb (2001); it has been completed and adapted by the authors. 
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Table 3. Selected results from national reviews of employment policies - EES, impact evaluation 
(focus Pillar I: employability, in particular training measures) 


Country 

Type of measures 

Review methodology 

Spain 

Training for the unemployed: design and delive- 
ry of training are decentralised to regional autho- 
rities; main target groups: short-term unemploy- 
ed youth and adults. 

Econometric analysis of the impact of measures for 
participants: chance of having a job at the time of 
the analysis (hiring rate); chance of having a job at 
one point during the observation period (employ- 
ability rate); estimation techniques for comparing 
the treatment group against a control group. 

The 

Netherlands 

Overview of a wide range 
of training measures. 

Drawing on previous evaluation (in particular De 
Koning, 1998). 

Ireland 

ALMP under the national employment action 
plan: support to the unemployed at an early sta- 
ge of unemployment period mainly by providing 
a job, followed with guidance and counselling (in 
this process training may be offered). 

Administrative data and consultations with key play- 
ers of social and labour market fields. 

United Kingdom 

New Deal Programme: targeted at young people 
(1 8-24) once they reach six months of unemploy- 
ment and those over 25 once they have been un- 
employed for 18 months. New Deal 50 Plus is tar- 
geted at those unemployed aged 50 and over, who 
are eligible after six months of unemployment. 

Administrative data 

Austria 

Training programmes upgrading the skills of the 
unemployed and responding directly to employ- 
ers’ needs. 

Qualification improvement programme through 
public employment service. 

Micro-analysis of programme participants and 
non-participants, adjusting the gross findings with 
econometric techniques to calculate estimates of net 
impact. 

Estimation of macroeconomic effects of policies. 

Finland 

Thorough appraisal of all four pillars. 

Administrative data combined with other sources 
such as the labour force survey. Various manipu- 
lations (including quasi-experimental techniques). 
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Main results 


Training had positive effects for most participants and significantly improved the employability of certain sub-groups (in 
particular disadvantaged youths, disabled persons and ex-convicts). 

Employability rate offers a better indicator of the longer-term job prospects of participants. 


Effects of participation in training programme changed significantly throughout the 1 990s. Until 1 993, there were signifi- 
cant positive effects, especially for long-term unemployment. After 1 997, no significant effect of programme participation 
can be detected. 


Although the report reaffirms commitment to such measures to help alleviate social exclusion and labour shortages, it also 
questions whether they always work. It suggests that the probability of leaving unemployment is rather dependent on the 
individual’s capacity to compete in the labour market than on the support provided through ALMP. 


Reasonable signs of success with 57 % of sample in work (the majority from the younger age groups, have shorter un- 
employment spell and are women). 


Positive macro impact of participating in training programmes, when upgrading of skills of the unemployed corresponds 
directly to employer needs. Increase of aggregate employment by over 9 800 each year. Contribution of 0.32 % to GDP 
growth. 

Earnings potential of participants in the qualification improvement programme increased by EUR 2 870 per annum (in com- 
parison with a control group). 


Overall employment effect of training measures were modest for the individuals concerned. 

Labour-market training has a weak net impact on employment probability of participants (outcomes slightly better for 
those with lower than average employability at the start of the programme). 

Interview-based survey shows considerable social benefits for programme participants. 

The study concludes that training designed for immediate employment should be responsive to the economic cycle, being 
reduced when fewer jobs are available. Training that helps to increase skills should, on the contrary, respond inversely to 
the economic cycle. 


NB: Walsh and Parsons did not consider reports from the other Member Sates because of their lack of specific coverage of ALMP 
and attention to detail. 

Source: Walsh and Parsons, 2004 based on national reports prepared in the context of the 2002 Impact evaluation of the EES 
(http://europa.eu.int/comm/employment_social/employment_strategy/impact_en.htm). 
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Table 4. Summary of main finding of ESF evaluations, Objectives 1 , 3 and 4 


Objective 1: to develop regions which are currently underdeveloped 


net impact was greater when programmes concentrated on the most disadvantaged groups than when they were targe- 
ted to all the unemployed; 

in some cases, combining measures increased their net impact; 

programmes offering work experience achieve better results when combined with some form of skills training. 


Objective 3: to tackle long-term unemployment, promote equal opportunities, 

improve lifelong learning, encourage entrepreneurship and adaptability 
and improve the role of women in the workforce 


overall, gross employment impact ranged from 30 to 80 %; 
clients showed high levels of satisfaction with their programmes; 

the chances of finding work after a programme depended less on the beneficiaries’ personal characteristics than on the 
availability of jobs in the labour market and the type of project. 


Objective 4: to improve the qualifications and prospects of all those in employment 


some evaluations suggested that employers benefited more than workers from the supported activities; 
future skills needs were generally not identified and taken into account into project design; 

some of the training supported through projects was considered too general or, in some cases, not sufficiently transfera- 
ble between employers. 


Source: Walsh and Parsons (2004) on the basis on European Commission (2001). 


• exogenous circumstances affect the relative effectiveness of training con- 
siderably. 

Local circumstances, including prevailing labour market and adminis- 
trative conditions, also limit the degree of generality and transferability of 
evaluation results. Consequently, the recommendations that can be drawn 
from reviewing evaluation work lead to ‘identifying common denominators 
that hint best practices’ (Walsh and Parsons, 2004, p. 253). 

Additional factors render evaluation work highly complex and tend to 
limit the usability of evaluation results ( 5 ): 

• evaluation results are highly dependent on the methods and data used. 
Examples show that for one specific programme, different evaluations 
can draw different conclusions (Hujer et al., 2004); 

• usually evaluations of ALMP focus strictly on earnings and employment 
as positive outcomes. Accounting for non-intended results (including neg- 
ative ones) is not part of the evaluation scope; 

• policy-makers and evaluation commissioners want quick results; eval- 
uation methods themselves incline evaluators towards measuring 
short-term outcomes ( 6 ), and evaluation budgets are often limited; 

• ALMP programmes tend to be small-scale; hence, even if a small pro- 
gramme has significant positive results for participants, it does not mean 


( 5 ) Partly based on Martin and Grubb (2001, pp. 10-13). 

( 6 ) One fundamental evaluation problem is that the more time passes, the more difficult it is to 
follow a sample and to attribute observed effects to the intervention. 
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that it would remain cost-effective and produce the same kind of pos- 
itive outcomes - without generating adverse effects - if implemented 
on a larger scale (in number of participants or in geographical terms); 

• finally, the task of evaluators is all the more complex because ALMP 
change according to specific needs and local contexts; various types 
of programmes are implemented in parallel; the mix of programmes is 
continuously changing and passive and active measures are often used 
complementarily to one another. 

Below we make some recommendations on dimensions and methods 
for training evaluations in ALMP that should get more attention in future. 

Broadening the evaluation scope 

The focus of evaluation should be broadened to ( 7 ): 

• address the determinants of outcomes, compared across subgroups of 
the labour market; 

• be carried out over longer periods to see whether short-term effects on 
earnings and employment also hold in the long run and whether there 
are delayed effects ( 8 ); 

• focus on increased employment probability or reduced unemployment 
duration as outcome variables. Earnings and wages might be question- 
able outcome variables in the European context because welfare state 
and minimum wage regulations may result in distortions between em- 
ployment status and earnings; 

• extend the scope of effects (or side-effects) to non-material ones (in- 
creased self-worth, better health, wider social and behavioural gains, 
further learning experiences, etc.); 

• foresee from the outset, i.e. when an intervention is designed, that the 
information required for an evaluation is gathered in a continuous process; 

• attempt to better assess the structure, content and design of training 
courses as well as whether they can be adapted to different circum- 
stances, such as changing labour-market needs. 

Although broadening the evaluation scope following these recommen- 
dations would considerably complicate the evaluator’s task and would re- 
quire methodological developments, we consider it would provide a fuller 
and more adequate picture of a programme’s outcomes than is currently 
the case. 


( 7 ) Summarised compilation of the recommendations made by several authors: Fay (1 996, cit- 
ed in Walsh and Parsons, 2004), Martin and Grubb (2001 , p. 1 6), Walsh and Parsons (2004), 
Hujer et al. (2004), Grubb and Ryan (1999). 

( 8 ) This aspect is crucial because programmes are established to address systemic and endur- 
ing problems such as unemployment, social exclusion, transition from school to work, etc. 
These are problems that require long-term solutions. Therefore, whether the short-term re- 
sults of a programme (what is generally captured by evaluation) are sustainable must be in- 
vestigated. 
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Opening the black box: the question of ‘why’? 

While evaluations tell us a lot about what works, or not, they are less 
instructive on another equally important and related question: why? Why 
do certain programmes work for some groups and not for others? Why do 
some circumstances increase the likelihood that a programme works? Why 
do specific programme designs and characteristics work well, better or not 
at all? Finding answers to these questions is central to designing cost-ef- 
fective public training programmes. Formative evaluation and qualitative meth- 
ods are required to address these questions because they deepen our un- 
derstanding of the way an intervention operates and participants react to 
it (Descy and Tessaring, 2005, pp. 73-74). 

Exploring programme design and implementation can help determine 
if the causes of a programme’s (non-) effectiveness are endogenous (e.g. 
appropriateness and quality of the learning process) or exogenous (e.g. the 
availability of jobs corresponding to the skills developed in the pro- 
gramme). 

One important but often neglected aspect is the learning process. Eval- 
uations often tend to assume that the expected learning process happened 
and resulted in the skills the programme intended to develop. In addition, 
few studies are concerned with the most effective teaching environments 
and methods ( 9 ), failing to recognise that it is an important element of pro- 
gramme design. One cannot conclude on the (non-) effectiveness of a cer- 
tain type of training without establishing whether any learning, let alone qual- 
ity learning took place. 

Evaluations that examine only the final outcomes of programmes in eco- 
nomic terms, such as earnings or employment, or even in non-economic 
terms, such as better health or reduction in crime, assume that this is suf- 
ficient to decide whether a programme should be maintained, expanded or 
terminated. Improving the quality of a programme to improve its positive out- 
comes should also be considered a key evaluation goal. 

Towards a systemic approach for evaluating ALMP 

Evaluations of the type discussed so far are conducted in what can be 
called a ‘programme’ perspective (Grubb and Ryan, 1999): training in ALMP 
is designed and implemented in a defined period and then evaluated. These 
evaluations reinforce the tendency to design policies that are limited in scope 
and target population, rather than aiming at long-term institution building. 

Grubb and Ryan (1 999, p. 1 09) comment: ‘[...] a programme is conceived 
of as something that can be created relatively quickly, introduced among 
the other institutions of a society, evaluated as a discrete entity, expand- 
ed or contracted. We call this a “project” or “programme view”, because of 
its tendency to think of a VET programme as self-contained and independ- 


( 9 ) Limiting itself to comparing on-the-job and off-the-job training, e.g. pedagogic and didactics 
are not considered. 
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ent. Then, what is conventionally called “programme evaluation” assess- 
es the effect of the programme only, independently of any surrounding poli- 
cies and institutions. We contrast this with what we call a “systems” view’ 
They elaborate later on (ibid. p. 116): '[...] only rarely do the evaluation of 
VET programmes conceive of a larger system of programmes [...] a sys- 
tem that might develop slowly over time. [...] A corollary is that more ener- 
gy is put into developing new programmes and evaluating them - and then 
abandoning them or trying new approaches - rather than continuing to de- 
velop institutions over longer periods of time [...] This suggests that the en- 
tire evaluation enterprise, in its “programme” focus, is part of an incoher- 
ent and fragmented approach that is unlikely to lead to more effective VET 
policies overtime. A systems perspective would, on the other hand, encour- 
age thinking not about individual projects, but about widely available pro- 
grammes that are linked to one another and institutionalised'. 

The basic reason for adopting a systemic approach in policy design and 
consequently in evaluation work is that any policy will interact with the rest 
of the system (institutions, organisations and actors) and with other systems 
(such as production). In addition, various types of programmes, led in par- 
allel, also interact. A systems perspective emphasises the need for more 
coherence between the various training interventions, ALMP as well as oth- 
er social policies. 

Issues of internal consistency (is this programme complementing, con- 
flicting or cancelling other active or passive measures?) and external con- 
sistency (is this programme consistent with the institutions in place, e.g. does 
it consider signalling patterns in the labour market?) are to be considered 
in both programme design and evaluation ( 10 ). Close to these issues is trans- 
parency, i.e. the ability of workers and employers to understand the role of 
a specific programme in the vast range of learning opportunities offered in 
education, vocational training and various ALMP. Whether ALMP consid- 
er sufficiently and are responsive to labour market skill needs and employ- 
ment demand is an important dimension to systemic evaluations. 

Finally, in a system perspective it is worth asking whether a programme 
is linked to other learning opportunities. Training in ALMP mostly has a nar- 
row focus on skills. To allow individuals to achieve both the levels and va- 
rieties of competences necessary throughout their lives, one should ensure 
that each programme can potentially be linked to other programmes (Grubb 
and Ryan, 1999, p. 118). Recognising and validating learning results of train- 
ing in ALMP could constitute a step in this direction. 

In conclusion, systemic evaluations consider whether a programme is 
successful because of its ability to be repeated, articulated with other VET 
programmes, connected to employers' hiring practices, and otherwise re- 
lated to other established practices and institutions. Policy-makers and 


( 10 ) See Descy and Tessaring (2005, pp. 127-130) for a discussion on internal and external con- 
sistency in systems' evaluation. 
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evaluators should expand their views on the nature of useful interventions. 
Developing programmes and measures has to be seen in a context where 
they relate to one another, in addition to their individual effectiveness. This 
should also be reflected in evaluation designs. 


Conclusion: continuous improvement 
of employment measures and policies 

ALMP are established in all EU countries, training being one of their key 
measures. Demonstrating their efficiency and improving their quality 
should be a concern for policy-makers. 

Traditional evaluations of ALMP ‘compare’ explicit programme goals with 
measurable outcomes (mainly in terms of employment probability and earn- 
ings). They estimate the relative effectiveness and/or efficiency of differ- 
ent interventions. Most evaluations are quasi-experimental, with emphasis 
on econometric elaboration of programme outcomes. However, because 
traditional evaluations do not open the ‘black box’, their results are gener- 
ally limited in indicating possibilities for change and improvement of pro- 
gramme design and implementation. They concentrate on what works rather 
than trying to answer why something works or not. They also tend to neg- 
lect interactions between various policy interventions and their cumulative 
impact. 

Evidence-led public policy is more in line with a holistic evaluation, com- 
bining formative and summative approaches, thus collecting the breadth 
and scope of information needed to provide feedback on quality. ‘Some of 
the best studies are those that take a wider methodological perspective, cer- 
tainly using [quasi-] experimental approaches where feasible, but comple- 
menting this with the use of administrative data and more qualitative infor- 
mation on for example, processes and the perceptions of programme par- 
ticipants. It suggests that evaluation hitherto has been more “academical- 
ly” driven rather than “policy” driven, which is not so much a criticism of the 
former as a lack of proper attention in the latter’ (Walsh and Parsons, 2004, 
p. 252). 

Evidence-led policy-making requires information on the circumstances, 
both exogenous and endogenous, that lead to programme quality, positive 
outcomes and cost-effectiveness. These requirements go beyond common 
evaluation practices. In consequence, a systemic approach to evaluation 
should be developed alongside existing techniques. It should broaden the 
current perception of effectiveness by also focusing on the way programmes 
and measures interact with other interventions and existing institutions. By 
doing so, it provides better-founded information to decide whether an in- 
tervention should be expanded or repeated and can lead to further learn- 
ing opportunities. Policy evaluation should define the criteria and provide 
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supporting empirical evidence on which policy types and which policy mix 
promise superior solutions to a society’s problem. ■ 
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